Comparison of Loglinear and Logistic Regression Models for Detecting Changes in Proportions

Two statistical procedures were compared in terms of their ability to detect significant changes in item difficulty and item discrimination from pretest to national test administrations. These two procedures were a loglinear (LL) analysis and a logistic regression (LR) procedure. The results of this study showed that the two procedures were able to identify two items from an ACT Mathematics Usage test showing unstable difficulty and discrimination between two administrations. Both procedures yielded similar results in showing stability in the remaining items, although rank, correlations of the results from LL and LR showed some inconsistency when discrimination changes were assessed, probably due to the use of fewer ability (score) categories in LL versus LR. COMPARISON OF LOGLINEAR AND LOGISTIC REGRESSION MODELS FOR DETECTING CHANGES IN PROPORTIONS Whenever test items are administered more than one time to samples of examinees that are theoretically from the same population, we should be concerned about sources of variation of proportion-correct values or p from sample to sample. One source of variation could simply be random sampling and if this were the case, differences in p might prove to be almost negligible or at least nonsignificant most of the time. On the other hand, significantly large differences in p might occur between testing administrations if there had been major changes in the item itself, such as the dropping of a popular foil or the rewriting or editing of the item's stem or alternatives. These significant variations in certain item characteristics or parameters could have major ramifications on test construction. Usual test construction practice is to select items for inclusion in a test by specifying a target distribution for p and then to select a set of items that matches, as closely as possible, that target distribution. If individual item parameters change significantly between test administrations, the actual distributions of these item parameters may no longer match the target distribution. In some applications, such as computerized adaptive testing, items are administered continually over some period of time. Change in item parameters in such situations has been termed parameter drift (Rentz, 1978). In the present paper, the interest is in situations where the same set of test items is administered twice, to two different samples of examinees. However, the methodology discussed within this paper can easily be extended to more than two administrations of the test items. The specific situation under consideration is one in which possible changes in p might occur between pretest and national administrations of the items in the ACT Assessment Program. Item statistics obtained from pretest